On Statistical Methods for Safety Validation of Automated Vehicles

Automated vehicles (AVs) are expected to bring safer and more convenient transport in the future. Consequently, before introducing AVs at scale to the general public, the required levels of safety should be shown with evidence. However, statistical evidence generated by brute force testing using safety drivers in real traffic does not scale well. Therefore, more efficient methods are needed to evaluate if an AV exhibits acceptable levels of risk.This thesis studies the use of two methods to evaluate the AV\u27s safety performance efficiently. Both methods are based on assessing near-collision using threat metrics to estimate the frequency of actual collisions. The first method, called subset simulation, is here used to search the scenario parameter space in a simulation environment to estimate the probability of collision for an AV under development. More specifically, this thesis explores how the choice of threat metric, used to guide the search, affects the precision of the failure rate estimation. The result shows significant differences between the metrics and that some provide precise and accurate estimates.The second method is based on Extreme Value Theory (EVT), which is used to model the behavior of rare events. In this thesis, near-collision scenarios are identified using threat metrics and then extrapolated to estimate the frequency of actual collisions. The collision frequency estimates from different types of threat metrics are assessed when used with EVT for AV safety validation. Results show that a metric relating to the point where a collision is unavoidable works best and provides credible estimates. In addition, this thesis proposes how EVT and threat metrics can be used as a proactive safety monitor for AVs deployed in real traffic. The concept is evaluated in a fictive development case and compared to a reactive approach of counting the actual events. It is found that the risk exposure of releasing a non-safe function can be significantly reduced by applying the proposed EVT monitor

On Safety Validation of Automated Driving Systems using Extreme Value Theory

Autonomous vehicles are expected to bring safer and more convenient transports in the future. When the system in the vehicle takes care of the driving, the driver is free to spend time on other things. As the driver is no longer part of the loop and cannot be used as a fallback, the requirements that are put on safety and dependability of the system will be very high. To test the system in real traffic and measure the failure rate that leads to an accident will therefore not be feasible. However, due to the complexity of the system, it is still desirable to be able to test the safety on a complete system level.With the emergence of automated driving systems, the vehicles will be equipped with an array of sensors that gives a representation of the environment. This opens up the possibility to use more information to estimate how safe the system behaves in real traffic. Using an area of statistics called Extreme Value Theory, the frequency of near-collision can be extrapolated into a frequency of actual collisions.These near-collisions are measured using threat assessment methods that have been developed for active safety applications. In this thesis, two types of measures are evaluated to determine how well they can be used for extrapolation. From the results, it is clear that the measure relating to a point where a collision is unavoidable works better than the one relating to the actual collision.Furthermore, several methods for automatically fitting the extreme value model to the data are evaluated. The result shows that all tested methods work well where some methods put emphasis on the more extreme data, which can result in a difference of the inferences drawn. This suggests that the whole process has the possibility to be automated, which is necessary when performed repeatedly on multiple large data sets

Comparing Collision Threat Measures for Verification of Autonomous Vehicles using Extreme Value Theory

The verification of safety is expected to be one of the largest challenges in the commercialization of autonomous vehicles. Using traditional methods would require infeasible time and resources. Recent research has shown the possibility of using near-collisions in order to estimate the frequency of actual collisions using Extreme Value Theory. However, little research has been done on how the measure for determining the closeness to a collision affect the result of the estimation. This paper compares a collision-based measure against one that relates to an inevitable collision state. The result shows that using inevitable collision states is more robust and that more research needs to be made into measures of collision proximity

A probabilistic framework for collision probability estimation and an analysis of the discretization precision

This paper presents a probabilistic framework for collision probability estimation. The framework uses information about objects\u27 velocity and acceleration gathered from a larger real traffic data set in order to create a discrete Markov Chain model. This model is then used to predict other traffic participants motion in a given scenario and through this calculate the probability of a future collision. The framework is then analyzed with respect to potential errors that are created in the discretization process. Especially the errors related to the discrete velocity regions are investigated in more detail. The analysis is performed on a selection of critical scenarios from a larger data set in order to set scenario-based requirements of the state discretization resolution. In the end, there is a discussion about the implications for the collision probability estimate, as well as, suggested next steps in order to get a complete view of the precision of the estimate

Using Extreme Value Theory for Vehicle Level Safety Validation and Implications for Autonomous Vehicles

Much effort is put right now into how to make autonomous vehicles as capable as possible in order to be able to replace humans as drivers. Less focus is put into how to ensure that this transition happens in a safe way that we can put trust in. The verification of the extreme dependability requirements connected to safety is expected to be one of the largest challenges to overcome in the commercialization of autonomous vehicles. Using traditional statistical methods to validate complete vehicle safety would require the vehicle to cover extreme distances to show that collisions occur rare enough. However, recent research has shown the possibility of using near-collisions in order to estimate the frequency of actual collisions using Extreme Value Theory. To use this method, there is a need for a measure related to the closeness of a collision. This paper shows that the choice of this threat measure has a significant impact on the inferences drawn from the data. With the right measure, this method can be used to validate the safety of a vehicle. This, while keeping the validity high and the data required lower than the state of the art statistical methods

Validation of Collision Frequency Estimation Using Extreme Value Theory

There is a lot of focus right now on how to build an autonomous vehicle, which can handle all the situations that a human driver is experiencing. Less is done on how to ensure that these vehicles are safe enough to be released to the public. Using traditional statistical methods would require one to drive extensive distances without incidents to prove the safety to a sufficient degree. Recent research has shown the possibility of using near-collisions in order to estimate the frequency of actual collisions using Extreme Value Theory. In order to trust these estimations, the precision of these estimates needs to be validated. The results from a 250 000 km field test shows that the Extreme Value estimations are reasonable in relation to a crash statistics estimate for rear-end collisions. This further suggests that extreme value is a method that can be used to predict collision frequencies from data containing no collisions

On Automated Vehicle Collision Risk Estimation using Threat Metrics in Subset Simulation

This paper presents a method for accelerated evaluation of an automated driving function using the subset simulation method. The focus of the paper is to investigate how the evaluation is affected by the choice of metric that is used to steer the subset simulation towards failure. This is done by comparing the use of some common threat assessment metrics and see how close the estimated failure rate of a function gets to a Monte Carlo simulation reference. The scope of this comparison is an ACC function that is faced with a set of cutin scenarios. It is found that all investigated metrics provide results relatively close to the reference, but the metrics relating to a state where collision is deemed to be unavoidable proved a little better